[Workload Management] Add QueryGroupServiceSettings #15028

ruai0511 · 2024-07-30T22:21:43Z

Description

This change introduces the QueryGroup service settings which we will use to enforce node level resiliency as part of this RFC: #12342.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-07-30T22:33:44Z

❌ Gradle check result for f8e758d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-07-30T22:39:44Z

❌ Gradle check result for 5324ca5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-07-30T22:46:56Z

❌ Gradle check result for 07aff39: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

...ramework/src/main/java/org/opensearch/search/query_group/QueryGroupServiceSettingsTests.java

server/src/main/java/org/opensearch/search/query_group/QueryGroupServiceSettings.java

github-actions · 2024-07-30T23:48:41Z

✅ Gradle check result for ff9d123: SUCCESS

github-actions · 2024-07-31T00:35:23Z

✅ Gradle check result for 3fa208c: SUCCESS

codecov · 2024-07-31T00:36:25Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.89%. Comparing base (13163ab) to head (47b91d6).
Report is 2 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15028      +/-   ##
============================================
- Coverage     71.92%   71.89%   -0.03%     
- Complexity    63194    63198       +4     
============================================
  Files          5213     5214       +1     
  Lines        295884   295935      +51     
  Branches      42754    42754              
============================================
- Hits         212820   212777      -43     
- Misses        65584    65688     +104     
+ Partials      17480    17470      -10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-08-20T21:35:59Z

✅ Gradle check result for 20e56f1: SUCCESS

server/src/main/java/org/opensearch/wlm/WorkloadManagementSettings.java

CHANGELOG.md

server/src/main/java/org/opensearch/wlm/WorkloadManagementSettings.java

jainankitk · 2024-08-21T00:23:53Z

@ruai0511 - Can you create issue for documentation PR as well?

Signed-off-by: Ruirui Zhang <[email protected]>

github-actions · 2024-08-21T00:53:26Z

✅ Gradle check result for fcedaec: SUCCESS

github-actions · 2024-08-21T01:39:56Z

✅ Gradle check result for 47b91d6: SUCCESS

opensearch-trigger-bot · 2024-08-21T05:20:05Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-15028-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 01184decb897dcdfc1b1d36980c4fc4eeddb8e54
# Push it to GitHub
git push --set-upstream origin backport/backport-15028-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-15028-to-2.x.

ruai0511 · 2024-08-21T19:07:52Z

@ruai0511 - Can you create issue for documentation PR as well?
Opened an issue here: opensearch-project/documentation-website#8066

…ct#15028) Signed-off-by: Ruirui Zhang <[email protected]>

…ct#15028) Signed-off-by: Ruirui Zhang <[email protected]> * add QeryGroup Service tests Signed-off-by: Ruirui Zhang <[email protected]> * add PR to changelog Signed-off-by: Ruirui Zhang <[email protected]> * change the test directory Signed-off-by: Ruirui Zhang <[email protected]> * modify comments to be more specific Signed-off-by: Ruirui Zhang <[email protected]> * add test coverage Signed-off-by: Ruirui Zhang <[email protected]> * remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]> (cherry picked from commit 01184de)

…ct#15028) * add QeryGroup Service tests Signed-off-by: Ruirui Zhang <[email protected]> * add PR to changelog Signed-off-by: Ruirui Zhang <[email protected]> * change the test directory Signed-off-by: Ruirui Zhang <[email protected]> * modify comments to be more specific Signed-off-by: Ruirui Zhang <[email protected]> * add test coverage Signed-off-by: Ruirui Zhang <[email protected]> * remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]>

Signed-off-by: Ruirui Zhang <[email protected]> * add QeryGroup Service tests Signed-off-by: Ruirui Zhang <[email protected]> * add PR to changelog Signed-off-by: Ruirui Zhang <[email protected]> * change the test directory Signed-off-by: Ruirui Zhang <[email protected]> * modify comments to be more specific Signed-off-by: Ruirui Zhang <[email protected]> * add test coverage Signed-off-by: Ruirui Zhang <[email protected]> * remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]> (cherry picked from commit 01184de) Signed-off-by: Ruirui Zhang <[email protected]> Signed-off-by: Ankit Jain <[email protected]> Co-authored-by: Ankit Jain <[email protected]>

@mch2

* Optimize global ordinal includes/excludes for prefix matching (opensearch-project#14371) * Optimize global ordinal includes/excludes for prefix matching If an aggregration specifies includes or excludes based on a regular expression, and the regular expression has a finite expansion followed by .*, then we can optimize the global ordinal filter. Specifically, in this case, we can expand the matching prefixes, then include/exclude the range of global ordinals that start with each prefix. Signed-off-by: Michael Froh <[email protected]> * Add unit test Signed-off-by: Michael Froh <[email protected]> * Add changelog entry Signed-off-by: Michael Froh <[email protected]> * Improve test coverage Updated the unit test to be functionally equivalent, but it covers more of the regex logic. Signed-off-by: Michael Froh <[email protected]> * Improve test coverage Signed-off-by: Michael Froh <[email protected]> * Fix bug in exclude-only case with no doc values in segment Signed-off-by: Michael Froh <[email protected]> * Address comments from @mch2 Signed-off-by: Michael Froh <[email protected]> --------- Signed-off-by: Michael Froh <[email protected]> * Adding access to noSubMatches and noOverlappingMatches in Hyphenation… (opensearch-project#13895) * Adding access to noSubMatches and noOverlappingMatches in HyphenationCompoundWordTokenFilter Signed-off-by: Evan Kielley <[email protected]> * Add Changelog Entry Signed-off-by: Mohammad Hasnain Mohsin Rajan <[email protected]> * test: add hyphenation decompounder tests Signed-off-by: Mohammad Hasnain <[email protected]> * test: refactor tests Signed-off-by: Mohammad Hasnain <[email protected]> * test: reformat test files Signed-off-by: Mohammad Hasnain <[email protected]> * chore: add changelog entry for 2.X Signed-off-by: Mohammad Hasnain <[email protected]> * chore: remove 3.x changelog Signed-off-by: Mohammad Hasnain <[email protected]> * chore: commonify settingsarr Signed-off-by: Mohammad Hasnain <[email protected]> * chore: commonify settingsarr Signed-off-by: Mohammad Hasnain <[email protected]> * chore: linting Signed-off-by: Mohammad Hasnain <[email protected]> --------- Signed-off-by: Evan Kielley <[email protected]> Signed-off-by: Mohammad Hasnain Mohsin Rajan <[email protected]> Signed-off-by: Mohammad Hasnain <[email protected]> Co-authored-by: Evan Kielley <[email protected]> * Add Settings related to Workload Management feature (opensearch-project#15028) * add QeryGroup Service tests Signed-off-by: Ruirui Zhang <[email protected]> * add PR to changelog Signed-off-by: Ruirui Zhang <[email protected]> * change the test directory Signed-off-by: Ruirui Zhang <[email protected]> * modify comments to be more specific Signed-off-by: Ruirui Zhang <[email protected]> * add test coverage Signed-off-by: Ruirui Zhang <[email protected]> * remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]> * Update affiliation for @nknize. (opensearch-project#15322) Signed-off-by: dblock <[email protected]> * Add log when download completes with file size (opensearch-project#15224) Signed-off-by: Gaurav Bafna <[email protected]> * Support Filtering on Large List encoded by Bitmap (version update) (opensearch-project#15352) Signed-off-by: Andriy Redko <[email protected]> * Add support for index level slice count setting (opensearch-project#15336) Signed-off-by: Ganesh Ramadurai <[email protected]> * Adding allowlist setting for ingest-useragent and ingest-geoip processors (opensearch-project#15325) * Adding allowlist setting for user-agent, geo-ip and updated tests for ingest-common. Signed-off-by: Sarat Vemulapalli <[email protected]> * Remove duplicate test in ingest-common Signed-off-by: Sarat Vemulapalli <[email protected]> * Adding changelog Signed-off-by: Sarat Vemulapalli <[email protected]> --------- Signed-off-by: Sarat Vemulapalli <[email protected]> * Add Delete QueryGroup API Logic (opensearch-project#14735) * Add Delete QueryGroup API Logic Signed-off-by: Ruirui Zhang <[email protected]> * modify changelog Signed-off-by: Ruirui Zhang <[email protected]> * include comments from create pr Signed-off-by: Ruirui Zhang <[email protected]> * remove delete all Signed-off-by: Ruirui Zhang <[email protected]> * rebase and address comments Signed-off-by: Ruirui Zhang <[email protected]> * rebase Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]> * add UT coverage Signed-off-by: Ruirui Zhang <[email protected]> * [Star Tree] Lucene Abstractions for Star Tree File Formats (opensearch-project#15278) --------- Signed-off-by: Sarthak Aggarwal <[email protected]> * [Star tree] Changes to handle derived metrics such as avg as part of star tree mapping (opensearch-project#15152) --------- Signed-off-by: Bharathwaj G <[email protected]> * relaxing the join validation for nodes which have only store disabled but only publication enabled * relaxing the join validation for nodes which have only store disabled but only publication enabled Signed-off-by: Rajiv Kumar Vaidyanathan <[email protected]> --------- Signed-off-by: Michael Froh <[email protected]> Signed-off-by: Evan Kielley <[email protected]> Signed-off-by: Mohammad Hasnain Mohsin Rajan <[email protected]> Signed-off-by: Mohammad Hasnain <[email protected]> Signed-off-by: dblock <[email protected]> Signed-off-by: Gaurav Bafna <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Ganesh Ramadurai <[email protected]> Signed-off-by: Sarat Vemulapalli <[email protected]> Signed-off-by: Rajiv Kumar Vaidyanathan <[email protected]> Co-authored-by: Michael Froh <[email protected]> Co-authored-by: Mohammad Hasnain Mohsin Rajan <[email protected]> Co-authored-by: Evan Kielley <[email protected]> Co-authored-by: Ruirui Zhang <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> Co-authored-by: Gaurav Bafna <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: Ganesh Krishna Ramadurai <[email protected]> Co-authored-by: Sarat Vemulapalli <[email protected]> Co-authored-by: Sarthak Aggarwal <[email protected]> Co-authored-by: Bharathwaj G <[email protected]> Co-authored-by: Rajiv Kumar Vaidyanathan <[email protected]>

…ct#15028) * add QeryGroup Service tests Signed-off-by: Ruirui Zhang <[email protected]> * add PR to changelog Signed-off-by: Ruirui Zhang <[email protected]> * change the test directory Signed-off-by: Ruirui Zhang <[email protected]> * modify comments to be more specific Signed-off-by: Ruirui Zhang <[email protected]> * add test coverage Signed-off-by: Ruirui Zhang <[email protected]> * remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService Signed-off-by: Ruirui Zhang <[email protected]> * address comments Signed-off-by: Ruirui Zhang <[email protected]>

ruai0511 requested review from anasalkouz, andrross, ashking94, Bukhtawar, CEHENKLE, dblock, dbwiddis, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami and VachaShah as code owners July 30, 2024 22:21

ruai0511 added the backport 2.x Backport to 2.x branch label Jul 30, 2024

ruai0511 changed the title ~~[Workload Management] add QeryGroupServiceSettings~~ [Workload Management] Add QeryGroupServiceSettings Jul 30, 2024

ruai0511 changed the title ~~[Workload Management] Add QeryGroupServiceSettings~~ [Workload Management] Add QueryGroupServiceSettings Jul 30, 2024

kaushalmahi12 reviewed Jul 30, 2024

View reviewed changes

ruai0511 force-pushed the settings-PR branch from 0b47585 to 20e56f1 Compare August 20, 2024 20:42

ruai0511 requested a review from linuxpi as a code owner August 20, 2024 20:42

jainankitk reviewed Aug 20, 2024

View reviewed changes

server/src/main/java/org/opensearch/wlm/WorkloadManagementSettings.java Outdated Show resolved Hide resolved

jainankitk reviewed Aug 20, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

ruai0511 force-pushed the settings-PR branch from 20e56f1 to fcedaec Compare August 21, 2024 00:03

jainankitk reviewed Aug 21, 2024

View reviewed changes

server/src/main/java/org/opensearch/wlm/WorkloadManagementSettings.java Outdated Show resolved Hide resolved

jainankitk reviewed Aug 21, 2024

View reviewed changes

server/src/main/java/org/opensearch/wlm/WorkloadManagementSettings.java Outdated Show resolved Hide resolved

address comments

47b91d6

Signed-off-by: Ruirui Zhang <[email protected]>

ruai0511 force-pushed the settings-PR branch from fcedaec to 47b91d6 Compare August 21, 2024 00:45

jainankitk approved these changes Aug 21, 2024

View reviewed changes

jainankitk merged commit 01184de into opensearch-project:main Aug 21, 2024
33 of 35 checks passed

opensearch-trigger-bot bot added the backport-failed label Aug 21, 2024

ruai0511 mentioned this pull request Aug 21, 2024

[DOC] Add Documentation for WorkloadManagement Settings opensearch-project/documentation-website#8066

Open

4 tasks

ruai0511 added a commit to kaushalmahi12/OpenSearch that referenced this pull request Aug 21, 2024

Add Settings related to Workload Management feature (opensearch-proje…

d8217dc

…ct#15028) Signed-off-by: Ruirui Zhang <[email protected]>

ruai0511 mentioned this pull request Aug 21, 2024

Backport: Add Settings related to Workload Management feature (#15028) #15345

Merged

3 tasks

opensearch-ci-bot mentioned this pull request Sep 9, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteSplitIndexIT #14296

Open

BrewTestBot mentioned this pull request Sep 18, 2024

opensearch 2.17.0 Homebrew/homebrew-core#191052

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Workload Management] Add QueryGroupServiceSettings #15028

[Workload Management] Add QueryGroupServiceSettings #15028

ruai0511 commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

github-actions bot commented Jul 31, 2024

codecov bot commented Jul 31, 2024 •

edited

Loading

github-actions bot commented Aug 20, 2024

jainankitk commented Aug 21, 2024

github-actions bot commented Aug 21, 2024

github-actions bot commented Aug 21, 2024

opensearch-trigger-bot bot commented Aug 21, 2024

ruai0511 commented Aug 21, 2024

[Workload Management] Add QueryGroupServiceSettings #15028

[Workload Management] Add QueryGroupServiceSettings #15028

Conversation

ruai0511 commented Jul 30, 2024

Description

Related Issues

Check List

github-actions bot commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

github-actions bot commented Jul 31, 2024

codecov bot commented Jul 31, 2024 • edited Loading

Codecov Report

github-actions bot commented Aug 20, 2024

jainankitk commented Aug 21, 2024

github-actions bot commented Aug 21, 2024

github-actions bot commented Aug 21, 2024

opensearch-trigger-bot bot commented Aug 21, 2024

ruai0511 commented Aug 21, 2024

codecov bot commented Jul 31, 2024 •

edited

Loading